Skip to content

fix: shutdown should wait for in flight requests#4702

Merged
steve-chavez merged 1 commit intoPostgREST:mainfrom
mkleczek:graceful-shutdowns
Apr 20, 2026
Merged

fix: shutdown should wait for in flight requests#4702
steve-chavez merged 1 commit intoPostgREST:mainfrom
mkleczek:graceful-shutdowns

Conversation

@mkleczek
Copy link
Copy Markdown
Collaborator

@mkleczek mkleczek commented Mar 10, 2026

DISCLAIMER:
This commit was authored entirely by a human without the assistance of LLMs.

Fixes #4689

@mkleczek mkleczek force-pushed the graceful-shutdowns branch from 8cbf39a to ed86489 Compare March 10, 2026 11:52
@mkleczek mkleczek changed the title add: Graceful shutdown fix: shutdown should wait for in flight requests Mar 10, 2026
@mkleczek mkleczek marked this pull request as ready for review March 10, 2026 11:53
@mkleczek mkleczek requested a review from steve-chavez March 10, 2026 11:54
@mkleczek mkleczek force-pushed the graceful-shutdowns branch 4 times, most recently from 91774ab to 6972cc4 Compare March 10, 2026 14:53
Comment thread test/io/test_io.py Outdated
@mkleczek mkleczek force-pushed the graceful-shutdowns branch from 6972cc4 to 2fb5020 Compare March 10, 2026 20:00
@mkleczek mkleczek marked this pull request as draft March 11, 2026 13:40
@mkleczek
Copy link
Copy Markdown
Collaborator Author

mkleczek commented Mar 11, 2026

Marked it as draft because there is a problem caused by yesodweb/wai#853.
What that means is we have to put the hard limit on gracefulShutdownTimeout otherwise warp will not shutdown at all if there is a reverse proxy keeping connections open (ie. using HTTP keep-alives).
@steve-chavez we need to decide what this limit should be. Unfortunately, warp is going to wait exactly that long if there are any keep-alive connections, so this timeout cannot be too long.

@mkleczek
Copy link
Copy Markdown
Collaborator Author

Related: yesodweb/wai#1064

@mkleczek
Copy link
Copy Markdown
Collaborator Author

@steve-chavez @wolfgangwalther - this is dependent on warp fixing yesodweb/wai#853. I've raised a draft PR yesodweb/wai#1064 but don't know if or when it will land.

First commit in this PR is a change to our Haskell overlay pointing to warp version from the above PR.

We need to decide what to do with this one.

@steve-chavez
Copy link
Copy Markdown
Member

this is dependent on warp fixing yesodweb/wai#853. I've raised a draft PR yesodweb/wai#1064 but don't know if or when it will land.

Nice, looks like the maintainer is already replying.

First commit in this PR is a change to our Haskell overlay pointing to warp version from the above PR.
We need to decide what to do with this one.

Since replies are fast I think we should aim to merge upstream and not fork/vendor for now.

@mkleczek mkleczek force-pushed the graceful-shutdowns branch 2 times, most recently from bcf3e19 to 2ddd662 Compare March 12, 2026 20:35
@mkleczek mkleczek force-pushed the graceful-shutdowns branch 2 times, most recently from 07da7bf to af7084e Compare April 18, 2026 05:31
@wolfgangwalther
Copy link
Copy Markdown
Member

But there are cabal build failures caused by some mysterious compilation errors when building jose-jwt. I am not sure what's going on but new warp introduces crypton >= 1.1 dependency (indirectly) which might be incompatible with jose-jwt.

I believe these mysterious compilation errors should be fixed with #4825 - at least I had some of these and I assume they were the same as yours :)

Comment thread stack.yaml Outdated
Comment thread cabal.project.freeze Outdated
Comment thread postgrest.cabal Outdated
Comment thread nix/overlays/haskell-packages.nix
@Vlix
Copy link
Copy Markdown

Vlix commented Apr 19, 2026

I am not sure what's going on but new warp introduces crypton >= 1.1 dependency (indirectly) which might be incompatible with jose-jwt.

If you use crypton-x509-1.8.0, it should work with crypton < 1.1. (haven't tried building the new warp with crypton-x509-1.8.0, so I can't guarantee it works)

@mkleczek mkleczek force-pushed the graceful-shutdowns branch 3 times, most recently from 71108a8 to b0ea187 Compare April 20, 2026 14:07
@mkleczek mkleczek marked this pull request as ready for review April 20, 2026 14:12
Comment thread test/io/test_io.py
@steve-chavez
Copy link
Copy Markdown
Member

It also looks like newer warp introduces some increased memory usage.

I see the failures:

not ok 7 - POST /rpc/leak?columns=blob: with a json key of 50M the memory usage(77,356,040 bytes) is more than 73M
not ok 8 - POST /leak?columns=blob: with a json key of 50M the memory usage(77,239,768 bytes) is more than 73M
not ok 9 - PATCH /leak?id=eq.1&columns=blob: with a json key of 50M the memory usage(77,304,464 bytes) is more than 73M

@mkleczek I think it should be fine to correct these lines to 78M.

jsonKeyTest "50M" "POST" "/rpc/leak?columns=blob" "73M"
jsonKeyTest "50M" "POST" "/leak?columns=blob" "73M"
jsonKeyTest "50M" "PATCH" "/leak?id=eq.1&columns=blob" "73M"

Comment thread CHANGELOG.md
Comment thread test/io/test_io.py
@mkleczek mkleczek force-pushed the graceful-shutdowns branch 2 times, most recently from c77c429 to cbd4b46 Compare April 20, 2026 19:43
Upgraded warp to 3.4.13 which fixed yesodweb/wai#853
Changed interrupt handling so that instead of killing the main thread, listening sockets are closed which triggers warp graceful shutdown.
@mkleczek mkleczek force-pushed the graceful-shutdowns branch from cbd4b46 to 17fcc57 Compare April 20, 2026 19:48
@mkleczek
Copy link
Copy Markdown
Collaborator Author

@mkleczek I think it should be fine to correct these lines to 78M.

Done

Copy link
Copy Markdown
Member

@steve-chavez steve-chavez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 🚀 🚀

@steve-chavez steve-chavez merged commit baebacf into PostgREST:main Apr 20, 2026
44 of 45 checks passed
@postgrest-ci
Copy link
Copy Markdown

postgrest-ci Bot commented Apr 20, 2026

Backport failed for v14, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin v14
git worktree add -d .worktree/backport-4702-to-v14 origin/v14
cd .worktree/backport-4702-to-v14
git switch --create backport-4702-to-v14
git cherry-pick -x baebacf3db76594a7799d3f76adad359f3fa2e25

@steve-chavez
Copy link
Copy Markdown
Member

steve-chavez commented Apr 20, 2026

It's not possible to backport this because it depends on

There will be a showstopping error:

       error: postgresql_13 has been removed since it reached its EOL upstream

@taimoorzaeem taimoorzaeem added no-backport Considered unsafe to backport since it carries a risk of breakage and removed backport v14 labels Apr 21, 2026
@steve-chavez
Copy link
Copy Markdown
Member

WDYT of changing this one to a feat instead of fix, that way we add docs for it. Some projects have "graceful shutdown" documented like:

We can mention this is done by Warp.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-backport Considered unsafe to backport since it carries a risk of breakage

Development

Successfully merging this pull request may close these issues.

Graceful shutdown

5 participants